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Effectiveness of a system-level initiative to create developmental math 
pathways that help students succeed 

Nationwide, community colleges educate nearly 40 percent of undergraduates. More than 
two-thirds of these students take at least one developmental course, as do 4 in 10 students at 4- 
year institutions (Ganga, Mazzariello, & Edgecombe, 2018). Developmental education requires 
substantial investments with costs to students and their families estimated at $1.3 billion per 
year, and total institutional costs estimated at $7 billion (Ganga et al., 2018). Students and their 
families make these investments supported by loans of roughly $3,000 per developmental course 
or an average $380 million in federal student loan debt accumulated each year (Belfield & 
Bailey, 2017). In developmental education, mathematics is the most frequently assessed 
developmental need and may be the single largest academic barrier to increasing overall college 
graduation rates (Attewell, Lavin, Domina, & Levey, 2006; Bryk & Treisman, 2010; Valentine, 
Konstantopoulos, & Goldrick-Rab, 2017). 

In response to low success rates of students who take traditional remedial mathematics in 
college, increasingly states and institutions of higher education have made optional or eliminated 
traditional prerequisite remediation and moved toward innovative developmental programs. A 
survey published in 2018 by the Education Commission of the States shows that 21 states have 
“authorized the use of innovative developmental education instructional methods and 
interventions” (“Key takeaways”; Whinnery & Pompelia, 2018). These instructional methods 
often incorporate and blend elements of streamlining, in which remedial work is condensed or 
combined with college-level work, and alignment, in which quantitative requirements are 
organized around students’ majors; for example, replacing algebra-calculus with statistics 
(Logue, Douglas, & Watanabe-Rose, 2019). Blending elements of these approaches is 
corequisite remediation, in which students may take college-level statistics or calculus courses 
with academic support, as needed (Complete College America, 2016; Malbin, 2016). 
Researchers argue streamlining and alignment approaches may be more successful for improving 
students’ success in developmental math than traditional remedial approaches because taking 
college-level courses may be more motivating. Among students who intend a social science or 
humanities major, alignment approaches offer a statistic-based alternative which is less abstract 
and easier for many to understand than algebra (Logue et al., 2019; Logue, Watanabe-Rose, & 
Douglas, 2016). 

MMRI approach to developmental mathematics education 

In response to low success among students who place into developmental education, as 
well as the high costs incurred by these students and the state, 12 Maryland institutions working 
with the University System of Maryland (USM) engaged in a redesign effort to create a 
statistics-based developmental math pathway in 2016. University System of Maryland 
Chancellor William E. (“Brit”) Kirwan charged a statewide workgroup to address the challenge 
of developmental mathematics in Maryland. Building on decades of statewide mathematics 
reform efforts and with funds from a Department of Education First in the World (FITW) grant, 
campus teams took a comprehensive approach to designing alternative developmental math 


pathways that involved advisors, counselors, and faculty. The Maryland Mathematics Reform 
Initiative (MMRI) FITW grant (MMRI-FITW) sought to “develop, implement, and evaluate a 
statistics pathway to accelerate developmental students’ progress into credit-bearing 
postsecondary courses and help more of those students reach certificate or degree completion 
effectively and efficiently” (“Background” section, second paragraph; Shapiro, 2016). The 
main/primary goal of the MMRI-FITW grant was to shorten the time for students to progress 
through developmental math by designing transferable courses that build skills aligned with 
program requirements of most social science and humanities majors. As research shows, most 
students choose non-science, technology, engineering, and math (non-STEM) majors and among 
those who do, it is more common for students to transfer out of rather than in to these programs 
(Chen & Soldner, 2013). Since most majors do not require algebra-based calculus, the MMRI- 
FITW approach addresses the “disconnect” between the algebra-based content students are 
taught and the statistics most students need to be successful in non-STEM fields (“Theory of 
Action” section, first paragraph; Shapiro, 2016). In particular, the goal of the grant was to 
develop and rigorously evaluate an alternative to developmental algebra-based math that 
prepared students for the statistics-based math required by most programs of study in the social 
sciences and humanities. Although various approaches are currently in use, there are few 
rigorous studies of the effectiveness of various alternatives. 

Per guidelines provided by the MMRI-FITW project, participating campuses could use 
grant funds to create an alternative developmental math course to prepare students for statistics 
(Morgan, Hall, & Shapiro, 2019; Shapiro, 2016). At most institutions, the counterfactual or 
“business-as-usual” condition consisted of a sequence of algebra-based developmental math 
classes that relied on use of baseline math test scores to decide how many and which courses 
students should take. For example, on some campuses, lower-scoring students might be required 
to pass as many as three developmental classes as prerequisites prior to enrolling in a credit- 
bearing, algebra-based math class, regardless of whether calculus was a requirement of their 
chosen major. 

Treatment course design. On most campuses, institutions participating in the MMRI- 
FITW evaluation offered a series of developmental algebra-based (comparison) courses and one 
developmental statistics-based (treatment) course. Some teams divided treatment courses into 
modules to make it easier for students to skip content they already mastered (Morgan, Hall, et al., 
2019; Rutschow, 2018; Rutschow & Diamond, 2015; Schak, Metzger, Bass, McCann, & 
English, 2017). Most campuses created one-semester courses designed to address knowledge 
gaps instead of multicourse sequences. 

Policymakers hypothesized that the alternative statistics pathway would be an effective 
strategy to reduce costs and time associated with taking multiple developmental algebra-based 
courses before accumulating college credits and successfully completing a postsecondary degree 
for students who choose programs in the social sciences or humanities (Shapiro, 2016). This 
study examines the impact of the MMRI-FITW developmental statistics-based course on 


students’ performance in developmental and college-level math and continued college 
enrollment. 
Evaluation Questions 
This study examined the impact of the alternative developmental treatment course on 
students’ rates of passing developmental math, passing college-level credit-bearing math, 
and continuous enrollment and addressed the following confirmatory and exploratory research 
questions: 
Confirmatory Question 1: Were there differences between treatment and comparison 
students in the rate at which they passed developmental math? 
Exploratory: Among students passing developmental math, were there differences 
between treatment and comparison students in the number of attempts they made before 
passing? 
Confirmatory Question 2: Were there differences between treatment and comparison 
students in the rate at which they passed college-level credit-bearing math? 
Exploratory: Were there differences between treatment and comparison students in the 
rate at which they enrolled in college-level credit-bearing math? 
Exploratory: Among students passing college-level credit-bearing math, were there 
differences between treatment and comparison students in the numbers of unique 
attempts they made before passing? 
Confirmatory Question 3: Were there differences between treatment and comparison 
students in the rate at which they remained continuously enrolled and/or graduated? 
Exploratory: Among students not continuously enrolled, were there differences between 
treatment and comparison students whether they returned? 
In the next section, we describe the study design, sample selection, participating schools, 
method for creating matched treatment and comparison groups, and data collection. 


Methods 

Study design. With extant data from eight institutions, we used a quasi-experimental 
design and matched sample of 2,041 (Treatment, n = 748; Comparison, n = 1,293) students to 
compare outcomes among students in the statistics-based developmental course with outcomes 
among students in traditional algebra-based math over four semesters. Because students decided 
whether to take the treatment or the comparison course (and were not randomly assigned), we 
used a quasi-experimental design and selected a matched comparison group to estimate the effect 
of the treatment on these outcomes: success in developmental and college math, and continuous 
enrollment or graduation. 

Participating schools. Twelve institutions of higher education participated in the project. 
During the study, we removed four schools from the study sample because their developmental 
math course sequences did not yield distinguishable treatment and comparison pathways (e.g., 
some students took courses from both pathways), or because too few students in the treatment 
course met our eligibility criteria to ensure their confidentiality. The eight remaining schools 


represented a range of institutions and included five community colleges and three 4-year 
institutions. The sample contained both large and small institutions, with the largest serving 
approximately 19,000 undergraduates per year and the smallest serving fewer than 700. The 
institutions were in rural, suburban, and city settings. At four of the retained participating 
institutions, non-White students represented at least half of enrolled undergraduates. Project 
leaders from the university system office provided project direction and support, and they 
convened institutions annually to discuss progress, address barriers, and share lessons learned 
from implementation across campuses (Morgan, Feagin, & Shapiro, 2019). In addition, system 
leaders invited nonparticipating institutions across the state to participate in project-wide 
meetings as affiliated members. 

Sample selection. Participating schools provided evaluators with a list of students who 
met the study’s four criteria. To be included in the sample, students had to enroll in the new 
statistics-based developmental course or the last algebra-based developmental course in a 
sequence between summer 2017 and spring 2018. Students also had to be a social science or 
humanities major, undecided, and/or not pursuing a STEM-related credential. Finally, students 
had to have Pell grant eligibility data (as a proxy for socioeconomic status) and a continuous 
baseline score of math ability to establish group equivalence at baseline. ! 

Creating the analytic sample. To increase confidence that differences in outcomes were 
the result of the treatment and not a result of other differences between students in the treatment 
and comparison groups, we matched students using Pell grant status as a proxy for 
socioeconomic status (SES) and baseline math ability (using math test scores). After confirming 
with campus teams that we correctly specified the various developmental treatment and 
comparison course pathways, we selected comparable treatment and comparison students who 
enrolled in their last developmental course before transitioning to college-level math. As many 
comparison students had passed earlier developmental courses, it is not surprising that 
comparison students had higher baseline math scores than did treatment group students.” To 
address that difference, we used a matching procedure to select a comparison group of students 
whose baseline test scores were similar to the baseline test scores of students in the treatment 
group. 

To account for students taking different placement tests, we standardized test scores, 
converting them to z-scores and controlling for the type of test in our model. Next, we compared 
distributions of z-scores within each institution and removed students in either group scoring 
lower or higher than the lower scoring group—for example, removing all treatment students in 
an institution with lower baseline scores than the lowest scoring comparison student. Finally, to 
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* Baseline math scores are defined as the standardized score on tests used to place students into 
developmental math classes. This was typically, but not exclusively, the ACCUPLACER 
Algebra test. 


promote group equivalence, we conducted sequential random removal of students in the 
comparison group who scored above the mean. We removed a random sample of 250 
comparison students scoring above the mean for the comparison group and recalculated the 
mean. Groups were still not within the adjustable range after one round of trimming, so a second 
random sample of 250 above-average scoring comparison students were removed and the 
matching model was run again. After matching, the two groups were within an acceptable range 
regarding math ability and able to be adjusted by including baseline math scores in the regression 
models (1.e., difference less than .25 standard deviations; see Table 1). The two groups were not 
statistically significantly different regarding SES (i.e., Pell eligibility) either before or after 
matching on baseline math ability, so no further trimming of the sample was required.* 


Table 1 


Baseline math ability and SES, before and after matching 


Baseline math ability SES (e.g., Pell grant status) 
Before alter Before After 
(N=2,696) (N=2,041) (N=2,696) (N=2,041) 
Treatment: mean (SD) -.489 (.774) -.410 (.734) 542 (.494) .579 (.495) 
Comparison: mean (SD) .209 (1.01) -.190 (.856) 552 (.497) 574 (.496) 
Standardized difference 70 22 01 .004 
Difference: p-value p <.001 p <.001 p= .076 p= 386 


While we matched groups only on SES and baseline math ability, we included other 
demographic variables as covariates in the model. As shown in Table 2, the treatment and 
comparison groups were descriptively similar on many of these characteristics. We used all 
variables in Tables 1 and 2 as covariates in our regression models. 
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Table 2 


Characteristics of the Matched Analytic Sample (N=2,041) 


Cohort 
Cohort 1 (Summer/ Fall 2017) 
Cohort 2 (Winter/ Spring 2018) 
Gender 
Male 
Female 
Missing 
Ethnicity 
Hispanic 
Not Hispanic 
Missing 
Race 
African American/Black 
American Indian/Alaskan Native 
Asian 
Pacific Islander 
White 
More than one race students 


Missing race 


Treatment 
(N=748) 


% 


51 


49 


<l 


36 


Comparison 
(N=1,293) 


% 


57 


43 


64 


18 


74 


15 


15 


Treatment Comparison 


(N=748) (N=1,293) 
% % 
Age (< or > 24) 
<24 79 Wl 
>and = 24 21 pe 
Full-time status 
Full-time 59 50 
Part-time 4] 50 
Institutions 
Institution 1 20 10 
Institution 2 3 3 
Institution 3 8 2 
Institution 4 2 7 
Institution 5 41 63 
Institution 6 13 5 
Institution 7 1 5 
Institution 8 11 5 


Data collection. Each semester, we received a file from participating institutions 
containing information about students’ performance in the treatment or comparison class; their 
performance in a subsequent college-level math class that meets their program’s requirements; 
and whether they stayed continuously enrolled and/or graduated from their home institution. We 
also collected information about students’ demographic characteristics, baseline Pell grant status, 
and math test scores. At the end of the study, each institution confirmed whether students had 
transferred to or graduated from other institutions during the study using data from the National 
Student Clearinghouse (NSC). Importantly, the NSC was only able to provide transfer (not 
course) data. 


Results 

We conducted a series of student-level multiple regression analyses to determine if there 
were statistically significant differences between treatment and comparison students across 
outcome domains. Each regression included student-level data from all eight institutions, with a 
dummy code for institution included as a covariate. We conducted one multiple regression per 
research question using the same predictor variables, changing only the outcome variable for 
each model. Predictor variables included: treatment status, standardized baseline math test score, 
institution, gender, race, Hispanic ethnicity, non-traditional age student, full-time status, cohort 
(1.e., initially taking the course in summer/fall 2017 or winter/spring 2018 during our study), and 
interaction of test type by baseline test score. 

Progress through developmental math. Our results suggest that the new statistics-based 
developmental math course positively influenced whether students passed developmental math. 
Participating in a treatment class significantly predicted whether students passed their 
developmental math course (f= .083, p<.001). Seventy-seven percent of treatment students 
passed developmental math compared to 69% of comparison students (Figure 1).4 Most students 
in both the treatment and comparison groups who passed developmental math did so after one 
attempt (91% and 83%, respectively). Overall, treatment students needed fewer attempts to pass 
developmental math than did comparison group students (f= -.083, p<.001). 


Figure 1 


Percentage of Students Who Passed Developmental Math (Adjusted for Covariates) 


Treatment Comparison 
(N=748) (N=1,293) 


“ Percentages reported are regression adjusted. 


Enrollment and success in credit-bearing math. Students in the treatment class were 
more likely than comparison students to enroll in a credit-bearing math course (f= .141, p<.001). 
Forty-nine percent of treatment students enrolled in credit-bearing math, relative to 34% of 
comparison students (Figure 2). 


Figure 2 


Percentage of Students Enrolled In Credit-Bearing Math, by Treatment Status 


Treatment Comparison 
(N=748) (N=1,293) 


Once enrolled in a credit-bearing math course, treatment students successfully passed at 
about the same rate as did comparison students. Taking the treatment or comparison class did not 
predict whether a student passed college-level math (f= -.017, p=.601). Also, among those 
passing their credit-bearing math course, treatment and comparison students took a similar 
number of attempts to pass their credit-bearing math course (6= .012, p=.731). 

Continuous enrollment. Taking the treatment or comparison course did not significantly 
predict whether students remained continuously enrolled and/or graduated during the study (f= - 
.009, p=.711). Fifty-six percent of treatment students and 57% of comparison students remained 
enrolled or graduated prior to the end of the study. Of the students who took one or more 
semesters off, 25% of treatment and 32% of comparison group students re-enrolled in the final 
semester of the study. 

Limitations. Although we included graduation rates as an outcome in our analyses, we 
did not expect students to graduate in 2 years. When interpreting these results, readers should 
bear in mind that a longer study might yield higher graduation rates. Also, we did not have 
access to data about students’ course-taking history. Some students may have taken 
developmental math courses before our study began. In cases where we report how many times a 
student took a developmental math course to pass it, our count includes only courses attempted 
since that start of our study. However, this concern applies to only 300 students who transferred 
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during the study and affected the treatment and comparison groups equally, minimizing concerns 
about biasing the results. 
Conclusion 

Building on decades of system-wide initiatives focused on mathematics reform, 
Maryland’s MMRI-FITW grant successfully created an effective alternative for non-STEM 
majors who enter college underprepared to succeed in the statistics courses required by their 
programs of study. Students who took the developmental treatment course passed at higher rates, 
in less time, and were more likely to enroll in credit-bearing coursework that satisfied 
requirements of their intended programs of study than did comparable students enrolled in the 
final algebra-based developmental course in a sequence. Furthermore, outcomes did not differ 
for part-time and non-traditional (age 24+) students, suggesting robust results across subgroups 
of students. 

We found that, once enrolled, treatment students and comparison students performed 
similarly in credit-bearing math coursework and remained enrolled, graduated, or returned to 
school at similar rates. This suggests that the MMRI-FITW alternative math pathways—focused 
on preparing students who are non-STEM majors to succeed in college-level statistics courses— 
are an effective approach to support success among students from low-SES backgrounds, or 
among older, or part-time, students who may need a refresher after taking several years off. 

In the Maryland example, designing developmental statistics courses that prepare students to 
succeed in the math required by most majors seems a promising developmental mathematics 
reform approach. Additional studies are needed to establish evidence-based practices for placing 
and advising students who take developmental math courses and intend to earn a social science 
or humanities credential, since once they enroll these students are as or are more successful as 
similar students who take traditional algebra-based developmental courses. 

Given the substantial investment that students and families make in developmental 
courses (Ganga et al., 2018), we need additional cost-effectiveness studies (e.g., Belfield & 
Bailey, 2017) to ensure students are making sound financial decisions and not accumulating 
unnecessary student loan debt. With systemic disruptions caused by the COVID-19 pandemic, 
resulting in fewer students choosing to invest and enroll in college, supporting the success of 
students who do enroll is especially critical now more than ever. 
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